Incentive-Compatible Critical Values

Statistical hypothesis tests are a cornerstone of scientific research. The informative when their size is properly controlled, so the frequency rejecting true null hypotheses (type I error) stays below prespecified nominal level. Publication bias exaggerates test sizes, however. Since scientists can typically only publish results that reject hypothesis, they have incentive to continue conducting studies until attaining rejection. Such $p$-hacking takes many forms: from collecting additional data examining multiple regression specifications, all in search statistical significance. process inflates sizes above levels because critical values used determine rejection assume statistics constructed single study---abstracting $p$-hacking. This paper addresses problem by constructing compatible with scientists' behavior given incentives. We researchers conduct finding statistic exceeds value, or benefit an extra study falls cost. then solve for incentive-compatible value (ICCV). When ICCV rejection, readers be confident controlled at desired significance level, and researcher's response incentives delineated accounted for. allow among studies, ICCVs larger than classical values. Yet, broad range researcher behaviors beliefs, lie fairly narrow range.